Method and apparatus for video coding

ABSTRACT

Aspects of the disclosure provide methods and apparatuses for video encoding/decoding. In some examples, an apparatus for video decoding includes processing circuitry that determines, from a coded video bitstream, a portion of a block is subject to a zero-out operation based on a threshold number of samples being less than a number of samples in one of a height dimension or a width dimension of the block. The processing circuitry determines whether a mixture of a lossless coding mode and one or more lossy coding modes is allowed in the coded video bitstream. The processing circuitry reconstructs the block based on the one or more lossy coding modes in response to (i) the mixture of the lossless coding mode and the one or more lossy coding modes being allowed in the coded video bitstream and (ii) the portion of the block being subject to the zero-out operation.

INCORPORATION BY REFERENCE

This present disclosure is a continuation of U.S. patent applicationSer. No. 16/887,489, “METHOD AND APPARATUS FOR VIDEO CODING,” filed onMay 29, 2020, which claims the benefit of priority to U.S. ProvisionalApplication No. 62/854,944, “FURTHER MODIFICATIONS OF LOSSLESS CODINGMODE IN VVC” filed on May 30, 2019. The disclosures of the priorapplications are incorporated by reference herein in their entirety.

TECHNICAL FIELD

The present disclosure describes embodiments generally related to videocoding.

BACKGROUND

The background description provided herein is for the purpose ofgenerally presenting the context of the disclosure. Work of thepresently named inventors, to the extent the work is described in thisbackground section, as well as aspects of the description that may nototherwise qualify as prior art at the time of filing, are neitherexpressly nor impliedly admitted as prior art against the presentdisclosure.

BACKGROUND

The background description provided herein is for the purpose ofgenerally presenting the context of the disclosure. Work of thepresently named inventors, to the extent the work is described in thisbackground section, as well as aspects of the description that may nototherwise qualify as prior art at the time of filing, are neitherexpressly nor impliedly admitted as prior art against the presentdisclosure.

Video coding and decoding can be performed using inter-pictureprediction with motion compensation. Uncompressed digital video caninclude a series of pictures, each picture having a spatial dimensionof, for example, 1920×1080 luminance samples and associated chrominancesamples. The series of pictures can have a fixed or variable picturerate (informally also known as frame rate), of, for example 60 picturesper second or 60 Hz. Uncompressed video has significant bitraterequirements. For example, 1080p604:2:0 video at 8 bit per sample(1920×1080 luminance sample resolution at 60 Hz frame rate) requiresclose to 1.5 Gbit/s bandwidth. An hour of such video requires more than600 GBytes of storage space.

One purpose of video coding and decoding can be the reduction ofredundancy in the input video signal, through compression. Compressioncan help reduce the aforementioned bandwidth or storage spacerequirements, in some cases by two orders of magnitude or more. Bothlossless and lossy compression, as well as a combination thereof can beemployed. Lossless compression refers to techniques where an exact copyof the original signal can be reconstructed from the compressed originalsignal. When using lossy compression, the reconstructed signal may notbe identical to the original signal, but the distortion between originaland reconstructed signals is small enough to make the reconstructedsignal useful for the intended application. In the case of video, lossycompression is widely employed. The amount of distortion tolerateddepends on the application; for example, users of certain consumerstreaming applications may tolerate higher distortion than users oftelevision distribution applications. The compression ratio achievablecan reflect that: higher allowable/tolerable distortion can yield highercompression ratios.

A video encoder and decoder can utilize techniques from several broadcategories, including, for example, motion compensation, transform,quantization, and entropy coding.

Video codec technologies can include techniques known as intra coding.In intra coding, sample values are represented without reference tosamples or other data from previously reconstructed reference pictures.In some video codecs, the picture is spatially subdivided into blocks ofsamples. When all blocks of samples are coded in intra mode, thatpicture can be an intra picture. Intra pictures and their derivationssuch as independent decoder refresh pictures, can be used to reset thedecoder state and can, therefore, be used as the first picture in acoded video bitstream and a video session, or as a still image. Thesamples of an intra block can be exposed to a transform, and thetransform coefficients can be quantized before entropy coding. Intraprediction can be a technique that minimizes sample values in thepre-transform domain. In some cases, the smaller the DC value after atransform is, and the smaller the AC coefficients are, the fewer thebits that are required at a given quantization step size to representthe block after entropy coding.

Traditional intra coding such as known from, for example MPEG-2generation coding technologies, does not use intra prediction. However,some newer video compression technologies include techniques thatattempt, from, for example, surrounding sample data and/or metadataobtained during the encoding/decoding of spatially neighboring, andpreceding in decoding order, blocks of data. Such techniques arehenceforth called “intra prediction” techniques. Note that in at leastsome cases, intra prediction is only using reference data from thecurrent picture under reconstruction and not from reference pictures.

There can be many different forms of intra prediction. When more thanone of such techniques can be used in a given video coding technology,the technique in use can be coded in an intra prediction mode. Incertain cases, modes can have submodes and/or parameters, and those canbe coded individually or included in the mode codeword. Which codewordto use for a given mode/submode/parameter combination can have an impactin the coding efficiency gain through intra prediction, and so can theentropy coding technology used to translate the codewords into abitstream.

A certain mode of intra prediction was introduced with H.264, refined inH.265, and further refined in newer coding technologies such as jointexploration model (JEM), versatile video coding (VVC), and benchmark set(BMS). A predictor block can be formed using neighboring sample valuesbelonging to already available samples. Sample values of neighboringsamples are copied into the predictor block according to a direction. Areference to the direction in use can be coded in the bitstream or mayitself be predicted.

SUMMARY

Aspects of the disclosure provide methods and apparatuses for videoencoding/decoding. In some examples, an apparatus for video decodingincludes receiving circuitry and processing circuitry. For example, theprocessing circuitry decodes, from a coded video bitstream, a flag thatis indicative of whether applying a lossless coding mode. In responsethe flag being indicative of applying the lossless coding mode, theprocessing circuitry partitions a block into multiple blocks based onthe block having at least one side that is equal to or greater than athreshold size for zeroing out a portion of the block. Thus, videoquality loss due to zero out can be avoided. Then, the processingcircuitry reconstructs the multiple blocks respectively from the codedvideo bitstream based on the lossless coding mode.

In some embodiments, the flag is signaled in at least one of a videoparameter set (VPS), a sequence parameter set (SPS), a picture parameterset (PPS), a slice header, a tile header, and a tile group header. In anembodiment, the processing circuitry splits the block into a left blockand a right block of equal sizes based on a width of the block beingequal to or greater than the threshold size and a height of the blockbeing less than the threshold size. In another embodiment, theprocessing circuitry splits the block into a top block and a bottomblock of equal sizes based on the width of the block being less than thethreshold size and the height of the block being equal to or greaterthan the threshold size. In another embodiment, the processing circuitrysplits the block into a top left block, a top right block, a bottom leftblock and a bottom right block of equal sizes based on both of the widthand the height of the block being equal to or greater than the thresholdsize.

In some examples, the block is a coding block or coding unit. In someother examples, the block is a transform block or transform unit.

In some embodiments, the processing circuitry disables the losslesscoding mode to be applied on the block based on a mixture of thelossless coding mode and lossy coding modes being allowed in the codedvideo bitstream, and reconstructs the block from the coded videobitstream based on a lossy coding mode.

In some embodiments, the processing circuitry sets a maximum coding treeunit (CTU) size to be less than the threshold size to avoid the zero-outoperation in coding tree units.

In an embodiment, the flag is a first flag associated with the block andindicative of the lossless coding mode associated with the block, thefirst flag is different from a second flag for indicating a bypass oftransform and quantization operations on the block.

In another embodiment, the flag is associated with the block and isindicative of the lossless coding mode that includes a bypass oftransform and quantization operations on the block.

Aspects of the disclosure also provide a non-transitorycomputer-readable medium storing instructions which when executed by acomputer for video decoding cause the computer to perform the method forvideo decoding.

BRIEF DESCRIPTION OF THE DRAWINGS

Further features, the nature, and various advantages of the disclosedsubject matter will be more apparent from the following detaileddescription and the accompanying drawings in which:

FIG. 1 is a schematic illustration of a simplified block diagram of acommunication system (100) in accordance with an embodiment;

FIG. 2 is a schematic illustration of a simplified block diagram of acommunication system (200) in accordance with an embodiment;

FIG. 3 is a schematic illustration of a simplified block diagram of adecoder in accordance with an embodiment;

FIG. 4 is a schematic illustration of a simplified block diagram of anencoder in accordance with an embodiment;

FIG. 5 shows a block diagram of an encoder in accordance with anotherembodiment;

FIG. 6 shows a block diagram of a decoder in accordance with anotherembodiment;

FIGS. 7A and 7B show a CTU that is partitioned by using a blockpartitioning structure;

FIGS. 8A and 8B show examples of triple tree partitioning;

FIGS. 9A and 9B show a syntax table in picture parameter set;

FIGS. 10A-10C show some syntax table examples;

FIG. 11 shows a luma mapping with chroma scaling (LMCS) architecturefrom decoder's perspective;

FIG. 12 shows a table of coding unit syntax in some embodiments;

FIG. 13 shows another table of coding unit syntax in some embodiments;

FIG. 14 shows an example of a table of residual coding syntax of acoding block;

FIG. 15 shows an example of a modified video coding standard accordingto some embodiments;

FIG. 16 shows another example of a modified video coding standardaccording to some embodiments;

FIG. 17 shows a table of coding unit syntax for coding a coding unit insome examples;

FIGS. 18A-18B show a table of coding unit syntax for coding a codingunit in some embodiments;

FIGS. 19A-19B show a table of transform unit syntax in some examples;

FIG. 20 shows a flow chart outlining a process example according to someembodiments of the disclosure; and

FIG. 21 is a schematic illustration of a computer system in accordancewith an embodiment.

DETAILED DESCRIPTION OF EMBODIMENTS

I. Encoder and Decoder

FIG. 1 illustrates a simplified block diagram of a communication system(100) according to an embodiment of the present disclosure. Thecommunication system (100) includes a plurality of terminal devices thatcan communicate with each other, via, for example, a network (150). Forexample, the communication system (100) includes a first pair ofterminal devices (110) and (120) interconnected via the network (150).In the FIG. 1 example, the first pair of terminal devices (110) and(120) performs unidirectional transmission of data. For example, theterminal device (110) may code video data (e.g., a stream of videopictures that are captured by the terminal device (110)) fortransmission to the other terminal device (120) via the network (150).The encoded video data can be transmitted in the form of one or morecoded video bitstreams. The terminal device (120) may receive the codedvideo data from the network (150), decode the coded video data torecover the video pictures and display video pictures according to therecovered video data. Unidirectional data transmission may be common inmedia serving applications and the like.

In another example, the communication system (100) includes a secondpair of terminal devices (130) and (140) that performs bidirectionaltransmission of coded video data that may occur, for example, duringvideoconferencing. For bidirectional transmission of data, in anexample, each terminal device of the terminal devices (130) and (140)may code video data (e.g., a stream of video pictures that are capturedby the terminal device) for transmission to the other terminal device ofthe terminal devices (130) and (140) via the network (150). Eachterminal device of the terminal devices (130) and (140) also may receivethe coded video data transmitted by the other terminal device of theterminal devices (130) and (140), and may decode the coded video data torecover the video pictures and may display video pictures at anaccessible display device according to the recovered video data.

In the FIG. 1 example, the terminal devices (110), (120), (130) and(140) may be illustrated as servers, personal computers and smart phonesbut the principles of the present disclosure may be not so limited.Embodiments of the present disclosure find application with laptopcomputers, tablet computers, media players and/or dedicated videoconferencing equipment. The network (150) represents any number ofnetworks that convey coded video data among the terminal devices (110),(120), (130) and (140), including for example wireline (wired) and/orwireless communication networks. The communication network (150) mayexchange data in circuit-switched and/or packet-switched channels.Representative networks include telecommunications networks, local areanetworks, wide area networks and/or the Internet. For the purposes ofthe present discussion, the architecture and topology of the network(150) may be immaterial to the operation of the present disclosureunless explained herein below.

FIG. 2 illustrates, as an example for an application for the disclosedsubject matter, the placement of a video encoder and a video decoder ina streaming environment. The disclosed subject matter can be equallyapplicable to other video enabled applications, including, for example,video conferencing, digital TV, storing of compressed video on digitalmedia including CD, DVD, memory stick and the like, and so on.

A streaming system may include a capture subsystem (213), that caninclude a video source (201), for example a digital camera, creating forexample a stream of video pictures (202) that are uncompressed. In anexample, the stream of video pictures (202) includes samples that aretaken by the digital camera. The stream of video pictures (202),depicted as a bold line to emphasize a high data volume when compared toencoded video data (204) (or coded video bitstreams), can be processedby an electronic device (220) that includes a video encoder (203)coupled to the video source (201). The video encoder (203) can includehardware, software, or a combination thereof to enable or implementaspects of the disclosed subject matter as described in more detailbelow. The encoded video data (204) (or encoded video bitstream (204)),depicted as a thin line to emphasize the lower data volume when comparedto the stream of video pictures (202), can be stored on a streamingserver (205) for future use. One or more streaming client subsystems,such as client subsystems (206) and (208) in FIG. 2 can access thestreaming server (205) to retrieve copies (207) and (209) of the encodedvideo data (204). A client subsystem (206) can include a video decoder(210), for example, in an electronic device (230). The video decoder(210) decodes the incoming copy (207) of the encoded video data andcreates an outgoing stream of video pictures (211) that can be renderedon a display (212) (e.g., display screen) or other rendering device (notdepicted). In some streaming systems, the encoded video data (204),(207), and (209) (e.g., video bitstreams) can be encoded according tocertain video coding/compression standards. Examples of those standardsinclude ITU-T Recommendation H.265. In an example, a video codingstandard under development is informally known as Versatile Video Coding(VVC). The disclosed subject matter may be used in the context of VVC.

It is noted that the electronic devices (220) and (230) can includeother components (not shown). For example, the electronic device (220)can include a video decoder (not shown) and the electronic device (230)can include a video encoder (not shown) as well.

FIG. 3 shows a block diagram of a video decoder (310) according to anembodiment of the present disclosure. The video decoder (310) can beincluded in an electronic device (330). The electronic device (330) caninclude a receiver (331) (e.g., receiving circuitry). The video decoder(310) can be used in the place of the video decoder (210) in the FIG. 2example.

The receiver (331) may receive one or more coded video sequences to bedecoded by the video decoder (310); in the same or another embodiment,one coded video sequence at a time, where the decoding of each codedvideo sequence is independent from other coded video sequences. Thecoded video sequence may be received from a channel (301), which may bea hardware/software link to a storage device which stores the encodedvideo data. The receiver (331) may receive the encoded video data withother data, for example, coded audio data and/or ancillary data streams,that may be forwarded to their respective using entities (not depicted).The receiver (331) may separate the coded video sequence from the otherdata. To combat network jitter, a buffer memory (315) may be coupled inbetween the receiver (331) and an entropy decoder/parser (320) (“parser(320)” henceforth). In certain applications, the buffer memory (315) ispart of the video decoder (310). In others, it can be outside of thevideo decoder (310) (not depicted). In still others, there can be abuffer memory (not depicted) outside of the video decoder (310), forexample to combat network jitter, and in addition another buffer memory(315) inside the video decoder (310), for example to handle playouttiming. When the receiver (331) is receiving data from a store/forwarddevice of sufficient bandwidth and controllability, or from anisosynchronous network, the buffer memory (315) may not be needed, orcan be small. For use on best effort packet networks such as theInternet, the buffer memory (315) may be required, can be comparativelylarge and can be advantageously of adaptive size, and may at leastpartially be implemented in an operating system or similar elements (notdepicted) outside of the video decoder (310).

The video decoder (310) may include the parser (320) to reconstructsymbols (321) from the coded video sequence. Categories of those symbolsinclude information used to manage operation of the video decoder (310),and potentially information to control a rendering device such as arender device (312) (e.g., a display screen) that is not an integralpart of the electronic device (330) but can be coupled to the electronicdevice (330), as was shown in FIG. 3. The control information for therendering device(s) may be in the form of Supplemental EnhancementInformation (SEI messages) or Video Usability Information (VUI)parameter set fragments (not depicted). The parser (320) mayparse/entropy-decode the coded video sequence that is received. Thecoding of the coded video sequence can be in accordance with a videocoding technology or standard, and can follow various principles,including variable length coding, Huffman coding, arithmetic coding withor without context sensitivity, and so forth. The parser (320) mayextract from the coded video sequence, a set of subgroup parameters forat least one of the subgroups of pixels in the video decoder, based uponat least one parameter corresponding to the group. Subgroups can includeGroups of Pictures (GOPs), pictures, tiles, slices, macroblocks, CodingUnits (CUs), blocks, Transform Units (TUs), Prediction Units (PUs) andso forth. The parser (320) may also extract from the coded videosequence information such as transform coefficients, quantizer parametervalues, motion vectors, and so forth.

The parser (320) may perform an entropy decoding/parsing operation onthe video sequence received from the buffer memory (315), so as tocreate symbols (321).

Reconstruction of the symbols (321) can involve multiple different unitsdepending on the type of the coded video picture or parts thereof (suchas: inter and intra picture, inter and intra block), and other factors.Which units are involved, and how, can be controlled by the subgroupcontrol information that was parsed from the coded video sequence by theparser (320). The flow of such subgroup control information between theparser (320) and the multiple units below is not depicted for clarity.

Beyond the functional blocks already mentioned, the video decoder (310)can be conceptually subdivided into a number of functional units asdescribed below. In a practical implementation operating undercommercial constraints, many of these units interact closely with eachother and can, at least partly, be integrated into each other. However,for the purpose of describing the disclosed subject matter, theconceptual subdivision into the functional units below is appropriate.

A first unit is the scaler/inverse transform unit (351). Thescaler/inverse transform unit (351) receives a quantized transformcoefficient as well as control information, including which transform touse, block size, quantization factor, quantization scaling matrices,etc. as symbol(s) (321) from the parser (320). The scaler/inversetransform unit (351) can output blocks comprising sample values, thatcan be input into aggregator (355).

In some cases, the output samples of the scaler/inverse transform (351)can pertain to an intra coded block; that is: a block that is not usingpredictive information from previously reconstructed pictures, but canuse predictive information from previously reconstructed parts of thecurrent picture. Such predictive information can be provided by an intrapicture prediction unit (352). In some cases, the intra pictureprediction unit (352) generates a block of the same size and shape ofthe block under reconstruction, using surrounding already reconstructedinformation fetched from the current picture buffer (358). The currentpicture buffer (358) buffers, for example, partly reconstructed currentpicture and/or fully reconstructed current picture. The aggregator(355), in some cases, adds, on a per sample basis, the predictioninformation the intra prediction unit (352) has generated to the outputsample information as provided by the scaler/inverse transform unit(351).

In other cases, the output samples of the scaler/inverse transform unit(351) can pertain to an inter coded, and potentially motion compensatedblock. In such a case, a motion compensation prediction unit (353) canaccess reference picture memory (357) to fetch samples used forprediction. After motion compensating the fetched samples in accordancewith the symbols (321) pertaining to the block, these samples can beadded by the aggregator (355) to the output of the scaler/inversetransform unit (351) (in this case called the residual samples orresidual signal) so as to generate output sample information. Theaddresses within the reference picture memory (357) from where themotion compensation prediction unit (353) fetches prediction samples canbe controlled by motion vectors, available to the motion compensationprediction unit (353) in the form of symbols (321) that can have, forexample X, Y, and reference picture components. Motion compensation alsocan include interpolation of sample values as fetched from the referencepicture memory (357) when sub-sample exact motion vectors are in use,motion vector prediction mechanisms, and so forth.

The output samples of the aggregator (355) can be subject to variousloop filtering techniques in the loop filter unit (356). Videocompression technologies can include in-loop filter technologies thatare controlled by parameters included in the coded video sequence (alsoreferred to as coded video bitstream) and made available to the loopfilter unit (356) as symbols (321) from the parser (320), but can alsobe responsive to meta-information obtained during the decoding ofprevious (in decoding order) parts of the coded picture or coded videosequence, as well as responsive to previously reconstructed andloop-filtered sample values.

The output of the loop filter unit (356) can be a sample stream that canbe output to the render device (312) as well as stored in the referencepicture memory (357) for use in future inter-picture prediction.

Certain coded pictures, once fully reconstructed, can be used asreference pictures for future prediction. For example, once a codedpicture corresponding to a current picture is fully reconstructed andthe coded picture has been identified as a reference picture (by, forexample, the parser (320)), the current picture buffer (358) can becomea part of the reference picture memory (357), and a fresh currentpicture buffer can be reallocated before commencing the reconstructionof the following coded picture.

The video decoder (310) may perform decoding operations according to apredetermined video compression technology in a standard, such as ITU-TRec. H.265. The coded video sequence may conform to a syntax specifiedby the video compression technology or standard being used, in the sensethat the coded video sequence adheres to both the syntax of the videocompression technology or standard and the profiles as documented in thevideo compression technology or standard. Specifically, a profile canselect certain tools as the only tools available for use under thatprofile from all the tools available in the video compression technologyor standard. Also necessary for compliance can be that the complexity ofthe coded video sequence is within bounds as defined by the level of thevideo compression technology or standard. In some cases, levels restrictthe maximum picture size, maximum frame rate, maximum reconstructionsample rate (measured in, for example megasamples per second), maximumreference picture size, and so on. Limits set by levels can, in somecases, be further restricted through Hypothetical Reference Decoder(HRD) specifications and metadata for HRD buffer management signaled inthe coded video sequence.

In an embodiment, the receiver (331) may receive additional (redundant)data with the encoded video. The additional data may be included as partof the coded video sequence(s). The additional data may be used by thevideo decoder (310) to properly decode the data and/or to moreaccurately reconstruct the original video data. Additional data can bein the form of, for example, temporal, spatial, or signal noise ratio(SNR) enhancement layers, redundant slices, redundant pictures, forwarderror correction codes, and so on.

FIG. 4 shows a block diagram of a video encoder (403) according to anembodiment of the present disclosure. The video encoder (403) isincluded in an electronic device (420). The electronic device (420)includes a transmitter (440) (e.g., transmitting circuitry). The videoencoder (403) can be used in the place of the video encoder (203) in theFIG. 2 example.

The video encoder (403) may receive video samples from a video source(401) (that is not part of the electronic device (420) in the FIG. 4example) that may capture video image(s) to be coded by the videoencoder (403). In another example, the video source (401) is a part ofthe electronic device (420).

The video source (401) may provide the source video sequence to be codedby the video encoder (403) in the form of a digital video sample streamthat can be of any suitable bit depth (for example: 8 bit, 10 bit, 12bit, . . . ), any colorspace (for example, BT.601 Y CrCB, RGB, . . . ),and any suitable sampling structure (for example Y CrCb 4:2:0, Y CrCb4:4:4). In a media serving system, the video source (401) may be astorage device storing previously prepared video. In a videoconferencingsystem, the video source (401) may be a camera that captures local imageinformation as a video sequence. Video data may be provided as aplurality of individual pictures that impart motion when viewed insequence. The pictures themselves may be organized as a spatial array ofpixels, wherein each pixel can comprise one or more samples depending onthe sampling structure, color space, etc. in use. A person skilled inthe art can readily understand the relationship between pixels andsamples. The description below focuses on samples.

According to an embodiment, the video encoder (403) may code andcompress the pictures of the source video sequence into a coded videosequence (443) in real time or under any other time constraints asrequired by the application. Enforcing appropriate coding speed is onefunction of a controller (450). In some embodiments, the controller(450) controls other functional units as described below and isfunctionally coupled to the other functional units. The coupling is notdepicted for clarity. Parameters set by the controller (450) can includerate control related parameters (picture skip, quantizer, lambda valueof rate-distortion optimization techniques, . . . ), picture size, groupof pictures (GOP) layout, maximum motion vector search range, and soforth. The controller (450) can be configured to have other suitablefunctions that pertain to the video encoder (403) optimized for acertain system design.

In some embodiments, the video encoder (403) is configured to operate ina coding loop. As an oversimplified description, in an example, thecoding loop can include a source coder (430) (e.g., responsible forcreating symbols, such as a symbol stream, based on an input picture tobe coded, and a reference picture(s)), and a (local) decoder (433)embedded in the video encoder (403). The decoder (433) reconstructs thesymbols to create the sample data in a similar manner as a (remote)decoder also would create (as any compression between symbols and codedvideo bitstream is lossless in the video compression technologiesconsidered in the disclosed subject matter). The reconstructed samplestream (sample data) is input to the reference picture memory (434). Asthe decoding of a symbol stream leads to bit-exact results independentof decoder location (local or remote), the content in the referencepicture memory (434) is also bit exact between the local encoder andremote encoder. In other words, the prediction part of an encoder “sees”as reference picture samples exactly the same sample values as a decoderwould “see” when using prediction during decoding. This fundamentalprinciple of reference picture synchronicity (and resulting drift, ifsynchronicity cannot be maintained, for example because of channelerrors) is used in some related arts as well.

The operation of the “local” decoder (433) can be the same as of a“remote” decoder, such as the video decoder (310), which has alreadybeen described in detail above in conjunction with FIG. 3. Brieflyreferring also to FIG. 3, however, as symbols are available andencoding/decoding of symbols to a coded video sequence by an entropycoder (445) and the parser (320) can be lossless, the entropy decodingparts of the video decoder (310), including the buffer memory (315), andparser (320) may not be fully implemented in the local decoder (433).

An observation that can be made at this point is that any decodertechnology except the parsing/entropy decoding that is present in adecoder also necessarily needs to be present, in substantially identicalfunctional form, in a corresponding encoder. For this reason, thedisclosed subject matter focuses on decoder operation. The descriptionof encoder technologies can be abbreviated as they are the inverse ofthe comprehensively described decoder technologies. Only in certainareas a more detail description is required and provided below.

During operation, in some examples, the source coder (430) may performmotion compensated predictive coding, which codes an input picturepredictively with reference to one or more previously coded picture fromthe video sequence that were designated as “reference pictures.” In thismanner, the coding engine (432) codes differences between pixel blocksof an input picture and pixel blocks of reference picture(s) that may beselected as prediction reference(s) to the input picture.

The local video decoder (433) may decode coded video data of picturesthat may be designated as reference pictures, based on symbols createdby the source coder (430). Operations of the coding engine (432) mayadvantageously be lossy processes. When the coded video data may bedecoded at a video decoder (not shown in FIG. 4), the reconstructedvideo sequence typically may be a replica of the source video sequencewith some errors. The local video decoder (433) replicates decodingprocesses that may be performed by the video decoder on referencepictures and may cause reconstructed reference pictures to be stored inthe reference picture cache (434). In this manner, the video encoder(403) may store copies of reconstructed reference pictures locally thathave common content as the reconstructed reference pictures that will beobtained by a far-end video decoder (absent transmission errors).

The predictor (435) may perform prediction searches for the codingengine (432). That is, for a new picture to be coded, the predictor(435) may search the reference picture memory (434) for sample data (ascandidate reference pixel blocks) or certain metadata such as referencepicture motion vectors, block shapes, and so on, that may serve as anappropriate prediction reference for the new pictures. The predictor(435) may operate on a sample block-by-pixel block basis to findappropriate prediction references. In some cases, as determined bysearch results obtained by the predictor (435), an input picture mayhave prediction references drawn from multiple reference pictures storedin the reference picture memory (434).

The controller (450) may manage coding operations of the source coder(430), including, for example, setting of parameters and subgroupparameters used for encoding the video data.

Output of all aforementioned functional units may be subjected toentropy coding in the entropy coder (445). The entropy coder (445)translates the symbols as generated by the various functional units intoa coded video sequence, by lossless compressing the symbols according totechnologies such as Huffman coding, variable length coding, arithmeticcoding, and so forth.

The transmitter (440) may buffer the coded video sequence(s) as createdby the entropy coder (445) to prepare for transmission via acommunication channel (460), which may be a hardware/software link to astorage device which would store the encoded video data. The transmitter(440) may merge coded video data from the video coder (403) with otherdata to be transmitted, for example, coded audio data and/or ancillarydata streams (sources not shown).

The controller (450) may manage operation of the video encoder (403).During coding, the controller (450) may assign to each coded picture acertain coded picture type, which may affect the coding techniques thatmay be applied to the respective picture. For example, pictures oftenmay be assigned as one of the following picture types:

An Intra Picture (I picture) may be one that may be coded and decodedwithout using any other picture in the sequence as a source ofprediction. Some video codecs allow for different types of intrapictures, including, for example Independent Decoder Refresh (“IDR”)Pictures. A person skilled in the art is aware of those variants of Ipictures and their respective applications and features.

A predictive picture (P picture) may be one that may be coded anddecoded using intra prediction or inter prediction using at most onemotion vector and reference index to predict the sample values of eachblock.

A bi-directionally predictive picture (B Picture) may be one that may becoded and decoded using intra prediction or inter prediction using atmost two motion vectors and reference indices to predict the samplevalues of each block. Similarly, multiple-predictive pictures can usemore than two reference pictures and associated metadata for thereconstruction of a single block.

Source pictures commonly may be subdivided spatially into a plurality ofsample blocks (for example, blocks of 4×4, 8×8, 4×8, or 16×16 sampleseach) and coded on a block-by-block basis. Blocks may be codedpredictively with reference to other (already coded) blocks asdetermined by the coding assignment applied to the blocks' respectivepictures. For example, blocks of I pictures may be codednon-predictively or they may be coded predictively with reference toalready coded blocks of the same picture (spatial prediction or intraprediction). Pixel blocks of P pictures may be coded predictively, viaspatial prediction or via temporal prediction with reference to onepreviously coded reference picture. Blocks of B pictures may be codedpredictively, via spatial prediction or via temporal prediction withreference to one or two previously coded reference pictures.

The video encoder (403) may perform coding operations according to apredetermined video coding technology or standard, such as ITU-T Rec.H.265. In its operation, the video encoder (403) may perform variouscompression operations, including predictive coding operations thatexploit temporal and spatial redundancies in the input video sequence.The coded video data, therefore, may conform to a syntax specified bythe video coding technology or standard being used.

In an embodiment, the transmitter (440) may transmit additional datawith the encoded video. The source coder (430) may include such data aspart of the coded video sequence. Additional data may comprisetemporal/spatial/SNR enhancement layers, other forms of redundant datasuch as redundant pictures and slices, SEI messages, VUI parameter setfragments, and so on.

A video may be captured as a plurality of source pictures (videopictures) in a temporal sequence. Intra-picture prediction (oftenabbreviated to intra prediction) makes use of spatial correlation in agiven picture, and inter-picture prediction makes uses of the (temporalor other) correlation between the pictures. In an example, a specificpicture under encoding/decoding, which is referred to as a currentpicture, is partitioned into blocks. When a block in the current pictureis similar to a reference block in a previously coded and still bufferedreference picture in the video, the block in the current picture can becoded by a vector that is referred to as a motion vector. The motionvector points to the reference block in the reference picture, and canhave a third dimension identifying the reference picture, in casemultiple reference pictures are in use.

In some embodiments, a bi-prediction technique can be used in theinter-picture prediction. According to the bi-prediction technique, tworeference pictures, such as a first reference picture and a secondreference picture that are both prior in decoding order to the currentpicture in the video (but may be in the past and future, respectively,in display order) are used. A block in the current picture can be codedby a first motion vector that points to a first reference block in thefirst reference picture, and a second motion vector that points to asecond reference block in the second reference picture. The block can bepredicted by a combination of the first reference block and the secondreference block.

Further, a merge mode technique can be used in the inter-pictureprediction to improve coding efficiency.

According to some embodiments of the disclosure, predictions, such asinter-picture predictions and intra-picture predictions are performed inthe unit of blocks. For example, according to the HEVC standard, apicture in a sequence of video pictures is partitioned into coding treeunits (CTU) for compression, the CTUs in a picture have the same size,such as 64×64 pixels, 32×32 pixels, or 16×16 pixels. In general, a CTUincludes three coding tree blocks (CTBs), which are one luma CTB and twochroma CTBs. Each CTU can be recursively quadtree split into one ormultiple coding units (CUs). For example, a CTU of 64×64 pixels can besplit into one CU of 64×64 pixels, or 4 CUs of 32×32 pixels, or 16 CUsof 16×16 pixels. In an example, each CU is analyzed to determine aprediction type for the CU, such as an inter prediction type or an intraprediction type. The CU is split into one or more prediction units (PUs)depending on the temporal and/or spatial predictability. Generally, eachPU includes a luma prediction block (PB), and two chroma PBs. In anembodiment, a prediction operation in coding (encoding/decoding) isperformed in the unit of a prediction block. Using a luma predictionblock as an example of a prediction block, the prediction block includesa matrix of values (e.g., luma values) for pixels, such as 8×8 pixels,16×16 pixels, 8×16 pixels, 16×8 pixels, and the like.

FIG. 5 shows a diagram of a video encoder (503) according to anotherembodiment of the disclosure. The video encoder (503) is configured toreceive a processing block (e.g., a prediction block) of sample valueswithin a current video picture in a sequence of video pictures, andencode the processing block into a coded picture that is part of a codedvideo sequence. In an example, the video encoder (503) is used in theplace of the video encoder (203) in the FIG. 2 example.

In an HEVC example, the video encoder (503) receives a matrix of samplevalues for a processing block, such as a prediction block of 8×8samples, and the like. The video encoder (503) determines whether theprocessing block is best coded using intra mode, inter mode, orbi-prediction mode using, for example, rate-distortion optimization.When the processing block is to be coded in intra mode, the videoencoder (503) may use an intra prediction technique to encode theprocessing block into the coded picture; and when the processing blockis to be coded in inter mode or bi-prediction mode, the video encoder(503) may use an inter prediction or bi-prediction technique,respectively, to encode the processing block into the coded picture. Incertain video coding technologies, merge mode can be an inter pictureprediction submode where the motion vector is derived from one or moremotion vector predictors without the benefit of a coded motion vectorcomponent outside the predictors. In certain other video codingtechnologies, a motion vector component applicable to the subject blockmay be present. In an example, the video encoder (503) includes othercomponents, such as a mode decision module (not shown) to determine themode of the processing blocks.

In the FIG. 5 example, the video encoder (503) includes the interencoder (530), an intra encoder (522), a residue calculator (523), aswitch (526), a residue encoder (524), a general controller (521), andan entropy encoder (525) coupled together as shown in FIG. 5.

The inter encoder (530) is configured to receive the samples of thecurrent block (e.g., a processing block), compare the block to one ormore reference blocks in reference pictures (e.g., blocks in previouspictures and later pictures), generate inter prediction information(e.g., description of redundant information according to inter encodingtechnique, motion vectors, merge mode information), and calculate interprediction results (e.g., predicted block) based on the inter predictioninformation using any suitable technique. In some examples, thereference pictures are decoded reference pictures that are decoded basedon the encoded video information.

The intra encoder (522) is configured to receive the samples of thecurrent block (e.g., a processing block), in some cases compare theblock to blocks already coded in the same picture, generate quantizedcoefficients after transform, and in some cases also intra predictioninformation (e.g., an intra prediction direction information accordingto one or more intra encoding techniques). In an example, the intraencoder (522) also calculates intra prediction results (e.g., predictedblock) based on the intra prediction information and reference blocks inthe same picture.

The general controller (521) is configured to determine general controldata and control other components of the video encoder (503) based onthe general control data. In an example, the general controller (521)determines the mode of the block, and provides a control signal to theswitch (526) based on the mode. For example, when the mode is the intramode, the general controller (521) controls the switch (526) to selectthe intra mode result for use by the residue calculator (523), andcontrols the entropy encoder (525) to select the intra predictioninformation and include the intra prediction information in thebitstream; and when the mode is the inter mode, the general controller(521) controls the switch (526) to select the inter prediction resultfor use by the residue calculator (523), and controls the entropyencoder (525) to select the inter prediction information and include theinter prediction information in the bitstream.

The residue calculator (523) is configured to calculate a difference(residue data) between the received block and prediction resultsselected from the intra encoder (522) or the inter encoder (530). Theresidue encoder (524) is configured to operate based on the residue datato encode the residue data to generate the transform coefficients. In anexample, the residue encoder (524) is configured to convert the residuedata from a spatial domain to a frequency domain, and generate thetransform coefficients. The transform coefficients are then subject toquantization processing to obtain quantized transform coefficients. Invarious embodiments, the video encoder (503) also includes a residuedecoder (528). The residue decoder (528) is configured to performinverse-transform, and generate the decoded residue data. The decodedresidue data can be suitably used by the intra encoder (522) and theinter encoder (530). For example, the inter encoder (530) can generatedecoded blocks based on the decoded residue data and inter predictioninformation, and the intra encoder (522) can generate decoded blocksbased on the decoded residue data and the intra prediction information.The decoded blocks are suitably processed to generate decoded picturesand the decoded pictures can be buffered in a memory circuit (not shown)and used as reference pictures in some examples.

The entropy encoder (525) is configured to format the bitstream toinclude the encoded block. The entropy encoder (525) is configured toinclude various information according to a suitable standard, such asthe HEVC standard. In an example, the entropy encoder (525) isconfigured to include the general control data, the selected predictioninformation (e.g., intra prediction information or inter predictioninformation), the residue information, and other suitable information inthe bitstream. Note that, according to the disclosed subject matter,when coding a block in the merge submode of either inter mode orbi-prediction mode, there is no residue information.

FIG. 6 shows a diagram of a video decoder (610) according to anotherembodiment of the disclosure. The video decoder (610) is configured toreceive coded pictures that are part of a coded video sequence, anddecode the coded pictures to generate reconstructed pictures. In anexample, the video decoder (610) is used in the place of the videodecoder (210) in the FIG. 2 example.

In the FIG. 6 example, the video decoder (610) includes an entropydecoder (671), an inter decoder (680), a residue decoder (673), areconstruction module (674), and an intra decoder (672) coupled togetheras shown in FIG. 6.

The entropy decoder (671) can be configured to reconstruct, from thecoded picture, certain symbols that represent the syntax elements ofwhich the coded picture is made up. Such symbols can include, forexample, the mode in which a block is coded (such as, for example, intramode, inter mode, bi-predicted mode, the latter two in merge submode oranother submode), prediction information (such as, for example, intraprediction information or inter prediction information) that canidentify certain sample or metadata that is used for prediction by theintra decoder (672) or the inter decoder (680), respectively, residualinformation in the form of, for example, quantized transformcoefficients, and the like. In an example, when the prediction mode isinter or bi-predicted mode, the inter prediction information is providedto the inter decoder (680); and when the prediction type is the intraprediction type, the intra prediction information is provided to theintra decoder (672). The residual information can be subject to inversequantization and is provided to the residue decoder (673).

The inter decoder (680) is configured to receive the inter predictioninformation, and generate inter prediction results based on the interprediction information.

The intra decoder (672) is configured to receive the intra predictioninformation, and generate prediction results based on the intraprediction information.

The residue decoder (673) is configured to perform inverse quantizationto extract de-quantized transform coefficients, and process thede-quantized transform coefficients to convert the residual from thefrequency domain to the spatial domain. The residue decoder (673) mayalso require certain control information (to include the QuantizerParameter (QP)), and that information may be provided by the entropydecoder (671) (data path not depicted as this may be low volume controlinformation only).

The reconstruction module (674) is configured to combine, in the spatialdomain, the residual as output by the residue decoder (673) and theprediction results (as output by the inter or intra prediction modulesas the case may be) to form a reconstructed block, that may be part ofthe reconstructed picture, which in turn may be part of thereconstructed video. It is noted that other suitable operations, such asa deblocking operation and the like, can be performed to improve thevisual quality.

It is noted that the video encoders (203), (403), and (503), and thevideo decoders (210), (310), and (610) can be implemented using anysuitable technique. In an embodiment, the video encoders (203), (403),and (503), and the video decoders (210), (310), and (610) can beimplemented using one or more integrated circuits. In anotherembodiment, the video encoders (203), (403), and (403), and the videodecoders (210), (310), and (610) can be implemented using one or moreprocessors that execute software instructions.

II. Lossless Coding Mode

Aspects of the disclosure provide techniques for use in lossless codingmode.

Generally, a block partitioning structure is used duringencoding/decoding. In some examples, the block partitioning structure isreferred to as a coding tree. In an example (e.g., HEVC), the codingtree can have a quad tree structure with each split splitting a largersquare block into four smaller square blocks. In some examples,according to the quad tree structure, a coding tree unit (CTU) is splitinto coding units (CUs) to adapt to various local characteristics. Adecision on whether to code a picture area using an inter-picture(temporal) or intra-picture (spatial) prediction is made at CU level.Each CU can be further split into one, two, or four prediction units(PUs) according to a PU splitting type. Inside one PU, a same predictionprocess is applied and relevant information is transmitted to a decoderon a PU basis.

After obtaining a residual block by applying a prediction process basedon the PU splitting type, a CU can be partitioned into transform units(TUs) according to another quad tree structure. In the example of HEVC,there are multiple partition conceptions including CU, PU, and TU. Insome embodiments, a CU or a TU can only be square shape, while a PU maybe square or rectangular shape. In some embodiments, one coding blockmay be further split into four square sub-blocks, and transform isperformed on each sub-block, i.e., TU. Each TU can be further splitrecursively into smaller TUs using a quad tree structure which is calledresidual quad tree (RQT).

At a picture boundary, in some embodiments, implicit quad tree split canbe employed so that a block will keep quad tree splitting until the sizefits the picture boundary.

In some examples, a block partitioning structure can use quad tree plusbinary tree (QTBT) block partitioning structure. The QTBT structure canremove the concepts of multiple partition types (the CU, PU and TUconcepts), and supports more flexibility for CU partition shapes. In theQTBT block partitioning structure, a CU can have either a square orrectangular shape.

FIG. 7A shows a CTU (710) that is partitioned by using a QTBT blockpartitioning structure (720) shown in FIG. 7B. The CTU (710) is firstpartitioned by a quad tree structure. The quad tree leaf nodes arefurther partitioned by a binary tree structure or a quad tree structure.There can be two splitting types, symmetric horizontal splitting (e.g.,labeled as “0” in the QTBT block partitioning structure (720)) andsymmetric vertical splitting (e.g., labeled as “1” in the QTBT blockpartitioning structure (720)), in the binary tree splitting. The leafnodes without further splitting are called CUs that can be used forprediction and transform processing without any further partitioning.Accordingly, CU, PU and TU have the same block size in the QTBT blockpartitioning structure.

In some embodiments, a CU can include coding blocks (CBs) of differentcolor components. For example, one CU contains one luma CB and twochroma CBs in the case of P and B slices of the 4:2:0 chroma format. ACU can include a CB of a single color component. For example, one CUcontains only one luma CB or just two chroma CBs in the case of Islices.

The following parameters are defined for the QTBT block partitioningscheme in some embodiments:

-   -   CTU size: the root node size of a quad tree, e.g. the same        concept as in HEVC.    -   MinQTSize: the minimum allowed quad tree leaf node size.    -   MaxBTSize: the maximum allowed binary tree root node size.    -   MaxBTDepth: the maximum allowed binary tree depth.    -   MinBTSize: the minimum allowed binary tree leaf node size.

In one example of the QTBT block partitioning structure, the CTU size isset as 128×128 luma samples with two corresponding 64×64 blocks ofchroma samples, the MinQTSize is set as 16×16, the MaxBTSize is set as64×64, the MinBTSize (for both width and height) is set as 4×4, and theMaxBTDepth is set as 4. The quad tree partitioning is applied to the CTUfirst to generate quad tree leaf nodes. The quad tree leaf nodes mayhave a size from 16×16 (i.e., the MinQTSize) to 128×128 (i.e., the CTUsize). If the leaf quad tree node is 128×128, it will not be furthersplit by the binary tree since the size exceeds the MaxBTSize (i.e.,64×64). Otherwise, the leaf quad tree node could be further partitionedby the binary tree. Therefore, the quad tree leaf node is also the rootnode for the binary tree and it has the binary tree depth as 0.

When the binary tree depth reaches MaxBTDepth (i.e., 4), no furthersplitting is considered. When the binary tree node has width equal toMinBTSize (i.e., 4), no further horizontal splitting is considered.Similarly, when the binary tree node has height equal to MinBTSize, nofurther vertical splitting is considered. The leaf nodes of the binarytree are further processed by prediction and transform processingwithout any further partitioning. In an embodiment, a maximum CTU sizeis 256×256 luma samples.

In FIGS. 7A and 7B, the solid lines indicate quad tree splitting anddotted lines indicate binary tree splitting. In each splitting (i.e.,non-leaf) node of the binary tree, one flag is signaled to indicatewhich splitting type (i.e., horizontal or vertical) is used. Forexample, 0 indicates a horizontal splitting and 1 indicates a verticalsplitting. For the quad tree splitting, there is no need to indicate thesplitting type since quad tree splitting can split a block bothhorizontally and vertically to produce 4 sub-blocks with an equal size.

In some embodiments, the QTBT block partitioning scheme supports theflexibility for the luma and chroma to have a separate QTBT blockpartitioning structure. For example, for P and B slices, the luma andchroma blocks in one CTU share the same QTBT block partitioningstructure. However, for I slices, the luma CTB is partitioned into CUsby a QTBT block partitioning structure, and the chroma blocks arepartitioned into chroma CUs by another QTBT block partitioningstructure. Thus, a CU in an I slice consists of a coding block of theluma component or coding blocks of two chroma components, and a CU in aP or B slice consists of coding blocks of all three color components.

In some embodiments, inter prediction for small blocks is restricted toreduce memory access of motion compensation. For example, bi-predictionis not supported for 4×8 and 8×4 blocks, and inter prediction is notsupported for 4×4 blocks.

Further, in some examples, a multi-type tree (MTT) block partitioningstructure is used. The MTT block partitioning structure is a moreflexible tree structure than the QTBT block partitioning structure. InMTT, in addition to quad tree partitioning and binary tree partitioning,horizontal center-side triple tree partitioning and vertical center-sidetriple tree partitioning can be used.

FIG. 8A shows an example of vertical center-side triple treepartitioning and FIG. 8B shows an example of horizontal center-sidetriple tree partitioning. Triple tree partitioning can complementquad-tree and binary-tree partitioning. For example, triple-treepartitioning is able to capture objects which locate in a block center,while quad-tree and binary-tree splits crossing block centers. The widthand height of partitions by triple trees are a power of 2 so that noadditional transform partition is needed.

According to some aspects of the disclosure, lossless coding mode isuseful and is supported in video standard, such as HEVC and the like. Insome application scenarios, such as medical imaging where loss ofinformation may result in diagnostic errors, any degradation of videoquality is not desired. Thus, a lossless coding approach can be enabledin the application scenarios.

In an embodiment, when a lossless coding mode is activated, transformand quantization can be bypassed, and one or more loop filter such asdeblocking filter can also be disabled. In an example, a flag, denotedby transquant_bypass_enable_flag, can be signaled in a picture parameterset (PPS). If a decoder receives such a flag and its value is signaledas 1, the decoder may bypass transform, quantization and loop filterprocessing.

FIGS. 9A and 9B show a syntax table (900) in picture parameter set. Inthe syntax table (900), a flag that is denoted astransquant_bypass_enable_flag (shown by (910)) can be used to enable ordisable lossless coding mode.

For example, when transquant_bypass_enable_flag is equal to 1, then a CUlevel bypass flag (indicating whether transform and quantization need tobe bypass at the CU level), that is denoted ascu_transquant_bypass_flag, can be present. Whentransquant_bypass_enable_flag is equal to 0, then the CU level bypassflag cu_transquant_bypass_flag is not present. For each CU, if thetransquant_bypass_enable_flag is true (e.g., “1”), the CU level bypassflag cu_transquant_bypass_flag can further be signaled for CUs toindicate whether transform, quantization and loop filter is disabled forcurrent CU. In an example, when the cu_transquant_bypass_flag issignaled as true (e.g., “1”), a syntax element indicating transform skipmode (TSM) (transform_skip_flag) is not signaled, and a sign data hidingmode is disabled.

In an example, when the cu_transquant_bypass_flag is equal to 1, thescaling and transform process and the in-loop filter process are need tobe bypassed. When the cu_transquant_bypass_flag is not present in anexample, the cu_transquant_bypass_flag can be inferred to be equal to 0.

In some embodiments, a specific residual coding mode can take theadvantage that Cb and Cr residuals appear to correlate inversely witheach other. The specific residual coding mode enables jointly coding ofchrominance residuals, and is referred to as joint chroma residualcoding mode (also referred to as joint Cb Cr residual (JCCR) codingmode). In JCCR coding mode, there is a single residual indicated for twochrominance blocks of a transform unit. The indicated residual is addedto the prediction block in the first channel (typically representing Cb)and deducted from the prediction block in the second channel (typicallyrepresenting Cr).

In an example, a joint chroma residual mode is indicated with a flag ina bitstream if the coded block flags (cbf) for both Cb and Cr are true.If the JCCR coding mode is activated, a single residual block isdecoded. The bitstream syntax and decoding process of joint residualblocks can be similar to that of regular Cb residuals in an example. Theresiduals of the Cr blocks are generated by negating the decoded jointresidual. As a single residual is used to represent residuals of twoblocks, it may often be desirable for this mode to use a QP lower thanwhat is used for separate coding of chrominance residuals. In anexample, a chroma QP offset of −1 was used for the joint mode and +1 wasused for separate chroma coding.

For the JCCR coding mode, several syntax elements at different codinglevel, such PPS level, tile level, block level and the like, are relatedto the JCCR coding mode.

FIG. 10A shows a syntax table (1010) for PPS raw byte sequence payload(RBSP) syntax. In FIG. 10A, a syntax element pps_joint_cbcr_qp_offsetshown by (1011) specifies the offset to the luma quantization parameterQp′_(Y) used for deriving Qp′_(CbCr). In an example, the value ofpps_joint_cbcr_qp_offset can be in the range of −12 to +12, inclusive.When ChromaArrayType is equal to 0, pps_joint_cbcr_qp_offset is not usedin the decoding process and decoders can ignore its value.

FIG. 10B shows a syntax table (1020) for general tile group headersyntax. In FIG. 10B, a syntax element tile_group_joint_cbcr_qp_offsetshown by (1021) specifies a difference to be added to the value ofpps_joint_cbcr_qp_offset when determining the value of the Qp′_(CbCr)quantization parameter. In an example, the value oftile_group_joint_cbcr_qp_offset can be in the range of −12 to +12,inclusive. When tile_group_joint_cbcr_qp_offset is not present, it isinferred to be equal to 0. The value ofpps_joint_cbcr_qp_offset+tile_group_joint_cbcr_qp_offset shall be in therange of −12 to +12, inclusive.

FIG. 10C shows a syntax table (1030) for residual coding syntax. In FIG.10C, the syntax element tu_joint_cbcr_residual[x0][y0] as shown in(1031), specifies whether indicated Cb residual is used to derive bothCb and Cr residuals. The array indices x0, y0 specify the location (x0,y0) of the top-left luma sample of the considered transform blockrelative to the top-left luma sample of the picture. In an example, thetu_joint_cbcr_residual[x0][y0] equal to 1 specifies that the indicatedCb residual is used to derive the Cr residual. Thetu_joint_cbcr_residual[x0][y0] equal to 0 specifies that Cr residual maybe present in the bitstream depending on other syntax elements. Whentu_joint_cbcr_residual[x0][y0] is not present, it is inferred to beequal to 0.

According to an aspect of the disclosure, joint chroma residual codingcan include several processes, such as a derivation process forquantization parameters, a scaling and transformation process, a scalingprocess for transform coefficients, and the like.

In an example, the derivation process for quantization parameters can beperformed as follows. The variables qP_(Cb), qP_(Cr) and qP_(CbCr) canbe derived as according to:

qPi _(Cb)=Clip3(−QpBdOffset_(C),69,Qp _(Y)+pps_cb_qp_offset+tile_group_cb_qp_offset)

qPi _(Cr)=Clip3(−QpBdOffset_(C),69,Qp _(Y)+pps_cr_qp_offset+tile_group_cr_qp_offset)

qPi _(CbCr)=Clip3(−QpBdOffset_(C),69,Qp _(Y)+pps_joint_cbcr_qp_offset+tile_group_joint_cbcr_qp_offset)

If ChromaArrayType is equal to 1, the variables qP_(Cb), qP_(Cr) andqP_(CbCr) are set equal to the value of Qp_(C) based on the index qPiequal to qPi_(Cb), qPi_(Cr) and qP_(CbCr), respectively. Otherwise, thevariables qP_(Cb), qP_(Cr) and qP_(CbCr) are set equal to Min(qPi, 63),based on the index qPi equal to qPi_(Cb), qPi_(Cr) and qP_(CbCr),respectively. The chroma quantization parameters for the Cb and Crcomponents, Qp′Cb and Qp′Cr, and joint Cb-Cr coding Qp′_(CbCr), arederived as follows:

Qp′ _(Cb) =qP _(Cb) +QpBdOffset_(C)

Qp′ _(Cr) =qP _(Cr) +QpBdOffset_(C)

Qp′ _(CbCr) =qP _(CbCr) +QpBdOffset_(C)

In an example, the scaling and transformation process can be performedas follows. The (nTbW)×(nTbH) array of residual samples resSamples canbe derived. For example, if cIdx is equal to 2 andtu_joint_cbcr_residual[xTbY][yTbY] is equal to 1, the residual samplesresSamples[x][y] with x=0 . . . nTbW−1, y=0 . . . nTbH−1 are derived asfollows:

resSamples[x][y]=−resSamplesCb[x][y]

where resSamplesCb[x][y] is the residual sample array generated for achrominance block with the transform block location (xTbY, yTbY) andcIdx equal to 1.

In an example, the scaling process for transform coefficients can beperformed as follows. The quantization parameter qP is derived asfollows. If cIdx is equal to 0, the following applies:

qP=Qp′ _(Y)

Otherwise, if cIdx is equal to 1 and tu_joint_cbcr_residual[xTbY][yTbY]is equal to 1, the following applies:

qP=Qp′ _(CbCr)

According to some aspects of the disclosure, a coding tool that isreferred to as luma mapping with chroma scaling (LMCS) is used.

In some embodiments, a coding tool LMCS is added as a processing blockbefore loop filters. LMCS can have two main components: 1) in-loopmapping of the luma component based on adaptive piecewise linear models;2) for the chroma components, luma-dependent chroma residual scaling isapplied.

FIG. 11 shows an LMCS architecture from decoder's perspective. Theblocks (1111, 1112, 1113) in FIG. 11 indicate where the processing isapplied in the mapped domain. The processing includes the inversequantization and inverse transform (1111), luma intra prediction (1113)and adding of the luma prediction together with the luma residual(1112). The blocks (1115, 1116, 1118, 1120-1124) in FIG. 11 indicatewhere the processing is applied in the original (i.e., non-mapped)domain. The processing includes loop filters (1115, or 1122) such asdeblocking, adaptive loop filter (ALF), and sample adaptive offset(SAO), motion compensated prediction (1118 or 1124), chroma intraprediction (1121), adding of the chroma prediction together with thechroma residual (1120), and storage of decoded pictures as referencepictures in decoded picture buffers (DPBs) (1116 or 1123). The blocks(1114, 1117, and 1119) in FIG. 11 are the LMCS functional blocks,including forward and inverse mapping of the luma signal (1114 and 1117)and a luma-dependent chroma scaling process (1119). In an example, LMCScan be enabled/disabled at the sequence level using a sequence parameterset (SPS) flag.

In some embodiments, luma mapping is performed with piecewise linearmodel. In some examples, the in-loop mapping of the luma componentadjusts the dynamic range of the input signal by redistributing thecodewords across the dynamic range to improve compression efficiency.Luma mapping makes use of a forward mapping function, FwdMap, and acorresponding inverse mapping function, InvMap. The FwdMap function issignaled using a piecewise linear model with 16 equal pieces. InvMapfunction does not need to be signaled and is instead derived from theFwdMap function.

In some examples, the luma mapping model is signaled at the tile grouplevel. A presence flag is signaled first. If luma mapping model ispresent in the current tile group, corresponding piecewise linear modelparameters are signaled. The piecewise linear model partitions the inputsignal's dynamic range into 16 equal pieces, and for each piece, linearmapping parameters of the piecewise linear model are expressed using thenumber of codewords assigned to that piece. Take 10-bit input as anexample. Each of the 16 pieces will have 64 codewords assigned to it bydefault. The signaled number of codewords is used to calculate thescaling factor and adjust the mapping function accordingly for thatpiece. At the tile group level, another LMCS enable flag is signaled toindicate if the LMCS process as depicted in FIG. 11 is applied to thecurrent tile group.

Each i-th piece, i=0 . . . 15, of the FwdMap piecewise linear model isdefined by two input pivot points InputPivot[ ] and two output (mapped)pivot points MappedPivot[ ].

The InputPivot[ ] and MappedPivot[ ] are computed as follows (assuming10-bit video):

(1) OrgCW=64

(2) For i=0:16, InputPivot[i]=i*OrgCW

(3) For i=0:16, MappedPivot[i] is calculated as follows:

-   -   MappedPivot[0]=0;    -   for(i=0; i<16; i++)        -   MappedPivot[i+1]=MappedPivot[i]+SignaledCW[i]            where SignaledCW[i] is the signaled number of codewords for            the i-th piece.

As shown in FIG. 11, for an inter-coded block, motion compensatedprediction is performed in the mapped domain. In other words, after themotion-compensated prediction block Y_(pred) is calculated based on thereference signals in the DPB, the FwdMap function is applied to map theluma prediction block in the original domain to the mapped domain,Y′_(pred)=FwdMap(Y_(pred)). For an intra-coded block, the FwdMapfunction is not applied because intra prediction is performed in themapped domain. After reconstructed block Y_(r) is calculated, the InvMapfunction is applied to convert the reconstructed luma values in themapped domain back to the reconstructed luma values in the originaldomain (Ŷ_(i)=InvMap(Y_(r))). The InvMap function is applied to bothintra- and inter-coded luma blocks.

The luma mapping process (forward and/or inverse mapping) can beimplemented using either look-up-tables (LUT) or using on-the-flycomputation. If LUT is used, then FwdMapLUT and InvMapLUT can bepre-calculated and pre-stored for use at the tile group level, andforward and inverse mapping can be simply implemented asFwdMap(Y_(pred)) FwdMapLUT[Y_(pred)] and InvMap(Y_(r))=InvMapLUT[Y_(r)],respectively. Alternatively, on-the-fly computation may be used. Takeforward mapping function FwdMap as an example. In order to figure outthe piece to which a luma sample belongs, the sample value is rightshifted by 6 bits (which corresponds to 16 equal pieces). Then, thelinear model parameters for that piece are retrieved and appliedon-the-fly to compute the mapped luma value. Let i be the piece index,a1, a2 be InputPivot[i] and InputPivot[i+1], respectively, and b1, b2 beMappedPivot[i] and MappedPivot[i+1], respectively. The FwdMap functionis evaluated as follows:

FwdMap(Y _(pred))=((b2−b1)/(a2−a1))*(Y _(pred) −a1)+b1

The InvMap function can be computed on-the-fly in a similar manner,except that conditional checks need to be applied instead of a simpleright bit-shift when figuring out the piece to which the sample valuebelongs, because the pieces in the mapped domain are not equal sized.

In some embodiments, luma-dependent chroma residual scaling can beperformed. Chroma residual scaling is designed to compensate for theinteraction between the luma signal and its corresponding chromasignals. Whether chroma residual scaling is enabled or not is alsosignaled at the tile group level. If luma mapping is enabled and if dualtree partition (also known as separate chroma tree) is not applied tothe current tile group, an additional flag is signaled to indicate ifluma-dependent chroma residual scaling is enabled or not. When lumamapping is not used, or when dual tree partition is used in the currenttile group, luma-dependent chroma residual scaling is disabled. Further,luma-dependent chroma residual scaling can be disabled for the chromablocks whose area is less than or equal to 4.

Chroma residual scaling depends on the average value of thecorresponding luma prediction block (for both intra- and inter-codedblocks). Denote avgY′ as the average of the luma prediction block. Thevalue of C_(ScaleInv) is computed in the following steps:

(1) Find the index Y_(Idx) of the piecewise linear model to which avgY′belongs based on the InvMap function.

(2) C_(ScaleInv)=cScaleInv[Y_(Idx)], where cScaleInv[ ] is apre-computed 16-piece LUT.

If the current block is coded as intra, combined inter and intraprediction (CIIP), or intra block copy (IBC, a.k.a. current picturereferencing or CPR) modes, avgY′ is computed as the average of theintra-, CIIP-, or IBC-predicted luma values; otherwise, avgY′ iscomputed as the average of the forward mapped inter predicted lumavalues (Y′_(pred) in FIG. 10). Unlike luma mapping, which is performedon the sample basis, C_(ScaleInv) is a constant value for the chromablock. With C_(ScaleInv) chroma residual scaling is applied as follows:

C _(ResScale) =C _(Res) *C _(Scale) =C _(Res) /C _(ScaleInv)  Encoderside:

C _(Res) =C _(ResScale) /C _(Scale) =C _(ResScale) *C_(ScaleInv)  Decoder side:

According to some aspects of the disclosure, block-based delta pulsecode modulation (BDPCM) can be used, and further residual in BDPCM modecan be quantized.

In some embodiments, BDPCM is employed that uses reconstructed samplesto predict the rows or columns line by line. For example, the signaledBDPCM direction indicates whether vertical or horizontal prediction isused. The reference pixels used are unfiltered samples. The predictionerror is quantized in the spatial domain. Pixels are reconstructed byadding the dequantized prediction error to the prediction.

As an alternative scheme to BDPCM, quantized residual domain BDPCM isused in some embodiments. The signaling and prediction directions usedcan be identical to BDPCM scheme. The intra prediction is done on theblock by sample copying in prediction direction (horizontal or verticalprediction) similar to intra prediction. The residual is quantized andthe delta between the quantized residual and its predictor (horizontalor vertical) quantized value is coded. This can be described as follows.

For a block of size M (rows)×N (cols), let r_(i,j), 0≤i≤M−1, 0≤j≤N−1 bethe prediction residual after performing intra prediction horizontally(copying left neighbor pixel value across the the predicted block lineby line) or vertically (copying top neighbor line to each line in thepredicted block) using unfiltered samples from above or left blockboundary samples. Let Q(r_(i,j)), 0≤i≤M−1, 0≤j≤N−1 denote the quantizedversion of the residual r_(i,j), where residual is difference betweenoriginal block and the predicted block values. Then the block DPCM isapplied to the quantized residual samples, resulting in modified M×Narray {tilde over (R)} with elements {tilde over (r)}_(i,j).

When vertical BDPCM is signaled:

${\overset{˜}{r}}_{i,j} = \left\{ {\begin{matrix}{{{Q\left( r_{i,j} \right)},}\ } & {{i = 0},} & {0 \leq j \leq \left( {N - 1} \right)} \\{{{Q\left( r_{i,j} \right)} - {Q\left( r_{{({i - 1})},j} \right)}},} & {{1 \leq i \leq \ \left( {M - 1} \right)},} & {0 \leq j \leq \ \left( {N - 1} \right)}\end{matrix}.} \right.$

For horizontal prediction, similar rules apply, and the residualquantized samples are obtained by

${\overset{˜}{r}}_{i,j} = \left\{ {\begin{matrix}{{{Q\left( r_{i,j} \right)},}\ } & {{0 \leq i \leq \left( {M - 1} \right)},} & {j = 0} \\{{{Q\left( r_{i,j} \right)} - {Q\left( {r_{i}}_{,{({j - 1})}} \right)}},} & {{0 \leq i \leq \ \left( {M - 1} \right)},} & {1 \leq j \leq \ \left( {N - 1} \right)}\end{matrix}.} \right.$

The residual quantized samples {tilde over (r)}_(i j) are sent to thedecoder.

On the decoder side, the above calculations are reversed to produceQ(r_(i,j)), 0≤i≤M−1, 0≤j≤N−1. For vertical prediction case,

Q(r _(i,j))=Σ_(k=0) ^(i) {tilde over (r)} _(k,j),0≤i≤(M−1),0≤j≤(N−1).

For horizontal case,

Q(r _(i,j))=Σ_(k=0) ^(j) {tilde over (r)} _(i,k),0≤i≤(M−1),0≤j≤(N−1).

The inverse quantized residuals, Q⁻¹(Q(r_(i,j))), are added to the intrablock prediction values to produce the reconstructed sample values.

In some examples (e.g., in VVC), for large transform block that has atleast one side larger than a threshold (e.g., 32), a portion of thecoefficients is coded. For example, a transform block has a size of 64×N(N is a positive integer), only the coefficients of the top-left32×min(N,32) region can be coded. In another example, a transform blockhas a size of N×64 (N is a positive integer), only the coefficients ofthe top-left min(N,32)×32 region can be coded. In the above twoexamples, the remaining higher frequency coefficients are not coded andconsidered as 0. Then, the coefficient coding for larger transform blockis not lossless.

Some aspects of the disclosure provide techniques for mathematicallylossless representation of the video signal for the video coding. Insome examples, coding tools which may create differences between inputand reconstructed video signal, such as multiple transform selection(MTS), secondary transform, chroma residual entropy coding, in-loopreshaper (or namely Luma mapping with chroma scaling, LMCS),quantized-domain residual differential pulse code modulation (RDPCM) anddependent quantization, and the like may be restricted for using inlossless coding mode.

In some examples, due to zero-out design, coefficient coding onlysupports N×32 and 32×N blocks, where N is an integer equal to or smallerthan 32, and lossless mode for larger blocks, such as 64×M and M×64blocks is undefined, where M is an integer equal to or smaller than 64.Thus, in some examples, some block partitioning techniques may depend onthe lossless coding mode.

In the following, the term block may be interpreted as a predictionblock, a coding block, or a coding unit, i.e. CU. Further, when sayinghigh-level syntax (HLS) element, it may refer to any of Video ParameterSet (VPS), Sequence Parameter Set (SPS), Picture Parameter Set (PPS),Slice header, Tile header, Tile group header. When saying CTU (codingtree unit, which is the largest CU size) header, it refers to syntaxelements signaled for each CTU, e.g., as header information. In thisdocument, when saying transform size, it may refer to the maximumtransform width and/height, or maximum transform unit area size.

In some embodiments, a flag can be signaled at CU level to indicatewhether a lossless coding mode is enabled for a respective coding block(or a respective coding unit).

FIG. 12 shows a table (1200) of coding unit syntax in some embodiments.The coding unit syntax includes a syntax element cu_lossless_mode_flagshown by (1210), that indicates whether a lossless coding mode isapplied to the coding block (or coding unit). In an example, thecu_lossless_mode_flag equal to 1 can specify that the lossless codingmode is enabled for the coding unit, while the cu_lossless_mode_flagequal to 0 can specify that the lossless coding mode is disabled for thecoding unit.

In some embodiments, a syntax element may be signaled to indicatewhether a lossless mode can be allowed for current video data associatedwith the syntax element. For example, the current data can be a tile, aslice, a tile group, a picture, a sequence, entire video. Accordingly,the syntax element can be signaled at a tile header, a slice header, atime group header, a PPS, a SPS, a video parameter set (VPS), or thelike. In one example, the syntax element indicating whether a losslessmode is enabled for current video data is denoted bylossless_mode_enable_flag. Semantics of the lossless_mode_enable_flag isas follows: lossless_mode_enable_flag equal to 1 specifies thatcu_lossless_mode_flag is present. lossless_mode_enable_flag equal to 0specifies that cu_lossless_mode_flag is not present.

FIG. 13 shows a table (1300) of coding unit syntax in some embodiments.As shown by (1310) in FIG. 13, the lossless_mode_enable_flag, whichindicates whether a CU level lossless mode flag presents, is checked.When the lossless_mode_enable_flag is true (e.g., having a value of 1),then the CU level lossless mode flag cu_lossless_mode_flag is presentfor example in the coded bitstream (also referred to as signaled in someexamples); otherwise, cu_lossless_mode_flag is not present in the codedbitstream for example.

In an embodiment, when a lossless coding mode flag cu_lossless_mode_flagis signaled with a value which indicates lossless coding mode is enabledfor a current CU, then the joint chroma residual coding mode is notapplied and a related indication flag tu_joint_cbcr_residual is notsignaled. The tu_joint_cbcr_residual can indicate whether the jointchroma residual coding mode is enabled or not.

FIG. 14 shows an example of a table (1400) of residual coding syntax ofa coding block. As shown by (1410) in FIG. 14, the presence of anindication flag tu_joint_cbcr_residual is partially based oncu_lossless_mode_flag being false.

In an embodiment, when a lossless coding mode flag cu_lossless_mode_flagis signaled with a value which indicates lossless coding mode is enabledfor a current CU, then LMCS is not applied no matter LMCS is enabled forthe current slice (or picture, or video sequence) or not.

As an example, FIG. 15 shows a text (1500) of a specification of a videocoding standard that is modified to accommodate implementation of alossless coding mode. A reconstruction process of a coding block basedon a slice_lmcs_enabled_flag is described in the text (1500). Theslice_lmcs_enabled_flag can be signaled in a slice header, and indicatewhether LMCS is enabled for a respective slice that includes the codingblock. At a paragraph (1501), a condition (1502) of“cu_lossless_mode_flag is equal to 1” is added. According to themodified paragraph (1501), when the condition (1502) is true (a losslessmode is enabled), a first operation (1503) can be performed, which isequivalent to disabling the LMCS. When the condition (1502) is false,the first operation (1503) or a second operation (1504) can be performeddepending on the value of slice_lmcs_enabled_flag.

FIG. 16 shows another text (1600) of a specification of a video codingstandard that is modified to accommodate implementation of a losslesscoding mode. A prediction process of CIIP mode for processing a codingblock is described in the text (1600). The prediction process can dependon a slice_lmcs_enabled_flag. At a paragraph (1601), a condition (1602)of “cu_lossless_mode_flag is equal to 0” is added. According to themodified paragraph (1601), when the condition (1602) is false (alossless mode is enabled), an operation (1603) can be skipped, which isequivalent to disabling the LMCS. When the condition (1602) is true, theoperation (1603) can be performed depending on the value ofslice_lmcs_enabled_flag.

In an embodiment, when a lossless coding mode flag cu_lossless_mode_flagis signaled with a value which indicates lossless coding mode is enabledfor a current CU, then residual domain BDPCM is not applied or signaled.

FIG. 17 shows a table (1700) of coding unit syntax for coding a codingunit in some examples. As shown by (1710) in FIG. 17, a high levelsyntax lossless_mode_enable_flag is checked. When thelossless_mode_enable_flag is true, it can be determined acu_lossless_mode_flag is signaled. Further, as shown by (1720) in FIG.17, the cu_lossless_mode_flag is used for verification. When thecu_lossless_mode_flag is true, it can be determined that a bdpcm_flag isnot signaled. Thus, the respective residual domain BDPCM can bedisabled. The bdpcm_flag can be a syntax element indicating whetherresidual domain BDPCM coding mode is enabled for the current codingunit.

In an embodiment, when a lossless coding mode flag cu_lossless_mode_flagis signaled with a value which indicates lossless coding mode is enabledfor current CU, some coding modes (e.g., MTS, transform skip (TS) mode,SBT, and secondary transform (ST)) are not applied or related syntaxelements are not signaled.

FIGS. 18A-18B show a table (1800) of coding unit syntax for coding acoding unit. As shown by (1810), a presence of a flag for SBT ispartially based on a cu_lossless_mode_flag. When thecu_lossless_mode_flag is true (indicating a lossless mode is enabled forthe coding unit), an indication flag cu_sbt_flag is not signaled. Thecu_sbt_flag can indicate if SBT is enabled for the coding unit.Accordingly, the SBT is not applied. Similarly, as shown by (1820), apresence of a flag for ST is partially based on thecu_lossless_mode_flag. When the cu_lossless_mode_flag is true, a syntaxelement, st_idx[x0][y0], is not signaled. The st_idx[x0][y0] is used torepresent a secondary transform index and indicate if secondarytransform (ST) mode is enabled for the coding unit. Accordingly, the STmode is not applied when the cu_lossless_mode_flag is true.

FIGS. 19A-19B show a table (1900) of transform unit syntax in someexamples. As shown by (1910), a presence of a flag for transform skipmode and a presence of a flag for MTS are partially based on thecu_lossless_mode_flag. When the cu_lossless_mode_flag is true(indicating a lossless mode is enabled for the coding unit), anindication flag transform_skip_flag[x0][y0], and a syntax elementtu_mts_idx[x0][y0] are not signaled. The transform_skip_flag[x0][y0] canindicate if TS mode is enabled for the coding unit. Thetu_mts_idx[x0][y0] can indicate an index of a selected transform if theMTS is enabled. Accordingly, the TS mode or the MTS is not applied whenthe cu_lossless_mode_flag is true.

In an embodiment, a high level syntax element, denoted byuse_lossless_mode_flag, can be signaled in a SPS, a PPS, a VPS, a sliceheader, a tile header or a tile group header, to indicate whether onlylossless coding modes are allowed for the current sequence, picture,video, slice, tile or tile group, respectively. When a lossless codingmode is used, lossy coding modes (or lossy coding tools) can be excludedfor encoding and decoding.

In an embodiment, the lossless coding mode flag cu_lossless_mode_flaghas slightly different indication from a flag cu_transquant_bypass_flag.For example, the flag cu_transquant_bypass_flag is used to indicatewhether to bypass transform and quantization, and the lossless codingmode flag is used to indicate whether to bypass operations that maycause loss of video quality, such as transform, quantization, SAO,deblocking, ALF, joint chroma residual coding, LMCS, TSM and the like.Thus, the lossless coding mode flag cu_lossless_mode_flag can besignaled separately in addition to the flag cu_transquant_bypass_flag.

In another embodiment, CU level flags cu_lossless_mode_flag andcu_transquant_bypass_flag can be merged as one single flag(cu_transquant_bypass_flag or cu_lossless_mode_flag) which is not onlyindicating whether transform and quantization is bypassed, but alsoindicating whether several other coding tools, such as SAO, deblocking,ALF, joint chroma residual coding, LMCS, TSM, and the like are enabledor not.

In some embodiments, an HLS syntax element for lossless mode, such asdenoted by use_lossless_mode_flag, is signaled in, for example, SPS,PPS, VPS, slice, tile or tile group, to indicate whether only losslesscoding modes are allowed for the current sequence, picture, video,slice, tile or tile group, respectively.

In an embodiment, the HLS syntax element is used similarly as the CUlevel flags cu_lossless_mode_flag in the above description, but in alarger scope corresponding to the high level. For example, when theuse_loseless_mode_flag is at a tile group level (or SPS, PPS, VPS,slice, tile and the like) and indicates that only lossless coding modesare allowed in the current tile group (or sequence, picture, video,slice, tile and the like), then, the lossless coding modes that whichexclude several coding tools (e.g., joint chroma residual coding mode,LMCS, residual domain BDPCM coding mode, MTS, transform skip (TS) mode,SBT, secondary transform (ST), and the like) that may cause videoquality loss can be used for encoding and decoding of the current tilegroup (or sequence, picture, video, slice, tile and the like).

According to some aspects of the disclosure, when the HLS syntax elementfor the lossless coding mode is indicative of only lossless coding modesare allowed, then certain size restrictions are applied during partitionto limit block (e.g., coding block, transform block and the like) sizes,thus lossless coding modes can be applied to the blocks with no videoquality loss.

In some embodiments, the lossless coding mode related HLS is signaledwith a value indicating the bitstream is only lossless coded. Then, ifthe current CU height and/or width is greater than or equal to thetransform size threshold (T) which always assume zero high-frequencycoefficients, i.e., 64 in VVC draft 5, then the CU will be split furtherwithout signaling.

In an embodiment, for a W×H CU (W denotes width, and H denotes height),if the CU width W is greater than or equal to T, but the CU height H isless than T, then CU is split using binary tree split and is splitvertically as two 0.5 W×H CUs.

In another embodiment, for a W×H CU, if the CU width W is less than T,but the CU height H is greater than or equal to T, then CU is splitusing binary tree split and is split horizontally as two W×0.5H CUs.

In another embodiment, for a W×H CU, if both the CU width W and CUheight H are greater than or equal to T, then CU is split using quadtreesplit as four 0.5 W×0.5H CUs.

In some embodiments, a mixture of lossless and lossy coding modes isallowed for the current bitstream. Then, if the current CU height and/orwidth is greater than or equal to the transform size threshold (T) whichalways assume zero high-frequency coefficients, i.e., 64 in VVC draft 5,then only lossy mode can be used on the current CU. In an example, theflag indicating whether lossless mode is used is not signaled but isinferred as false, which means lossless mode is not used on the currentCU.

In some embodiments, when the lossless coding mode related HLS syntax issignaled with a value indicating lossless coding mode can be can beused, then the CTU size is set as the maximum CU size which does notassume zero high-frequency coefficients, such as 32×32 in VVC draft 5.Thus, no CU will have size that may cause zero high-frequencycoefficients.

In some embodiments, when the lossless coding mode related HLS issignaled with a value indicating that lossless coding mode can be used,if the current CU height and/or width is greater than or equal to thetransform size threshold T which always assume zero high-frequencycoefficients, i.e., 64 in VVC draft 5, then the TU of the current CUwill be split further without signaling.

In an embodiment, for a CU, if the TU width (W) is greater than or equalto T, but the TU height (H) is not greater than or equal to T, then TUis split using binary tree split and is split vertically as two 0.5 W×HTUs.

In another embodiment, for a CU, if the TU width (W) is not greater thanor equal to T, but the TU height (H) is greater than or equal to T, thenTU is split using binary tree split and is split horizontally as twoW×0.5H TUs.

In another embodiment, for a CU, if both the TU width (W) and CU height(H) are greater than or equal to T, then TU is split using quadtreesplit as four 0.5 W×0.5H TUs.

III. Flowchart

FIG. 20 shows a flow chart outlining a process (2000) according to anembodiment of the disclosure. The process (2000) can be used in thereconstruction of a block, so to generate a prediction block for theblock under reconstruction. In various embodiments, the process (2000)are executed by processing circuitry, such as the processing circuitryin the terminal devices (110), (120), (130) and (140), the processingcircuitry that performs functions of the video encoder (203), theprocessing circuitry that performs functions of the video decoder (210),the processing circuitry that performs functions of the video decoder(310), the processing circuitry that performs functions of the videoencoder (403), and the like. In some embodiments, the process (2000) isimplemented in software instructions, thus when the processing circuitryexecutes the software instructions, the processing circuitry performsthe process (2000). The process starts at (S2001) and proceeds to(S2010).

At (S2010), a flag is decoded from a coded video bitstream. The flag isused to indicate whether applying a lossless coding mode in the codedvideo bitstream, such as on a block in the coded video bistream, in atile, in a tile group, in a slice, in a picture, in a sequence ofpictures, in a video, and the like.

At (S2020), when the flag is indicative of applying the lossless codingmode to a block, and the block has at least one side that is equal to orgreater than a threshold size for zeroing out a portion of the block,the block is partitioned into multiple smaller blocks with sides beingless than the threshold size. Thus, no portion of the block has beenzeroed out, and video quality loss due to zero out can be avoided. Thezero out operations can cause video quality loss and are avoided bysplitting the block into multiple smaller blocks.

In an embodiment, the width of the block is equal to or greater than thethreshold size and the height of the block is less than the thresholdsize splitting, then the block is split into a left block and a rightblock of equal sizes (e.g., two rectangular blocks).

In another embodiment, the width of the block is less than the thresholdsize and the height of the block is equal to or greater than thethreshold size, then the block is split into a top block and a bottomblock of equal sizes (e.g., two rectangular blocks).

In another embodiment, both of the width and the height of the blockbeing equal to or greater than the threshold size, then the block issplit into a top left block, a top right block, a bottom left block anda bottom right block of equal sizes (e.g., four rectangular blocks).

In some examples, the block can be a coding block. In some otherexamples, the block can be a transform block.

At (S2030), the multiple smaller blocks are reconstructed bsed on thelossless coding mode. Then the process proceeds to 52099 and terminates.

It is noted that, in some embodiments, a mixture of the lossless codingmode and lossy coding modes can be allowed in the coded video bitstream,and then the lossless coding mode can be disabled for applying on theblock. Then, the block can be reconstructed based on a lossy codingmode.

It is also noted that, in some embodiment, a maximum coding tree unit(CTU) size can be set to be less than the threshold size to avoid thezero-out operation in coding tree units.

It is also noted that, in an embodiment, when the flag for the losslesscoding mode is a CU level flag, the flag can be signaled separately fromanother flag that is used to indicate bypassing transform andquantization. In another embodiment, the flag for the lossless codingmode can be merged with the other flag for bypassing transform andquantization.

IV. Computer System

The techniques described above, can be implemented as computer softwareusing computer-readable instructions and physically stored in one ormore computer-readable media. For example, FIG. 21 shows a computersystem (2100) suitable for implementing certain embodiments of thedisclosed subject matter.

The computer software can be coded using any suitable machine code orcomputer language, that may be subject to assembly, compilation,linking, or like mechanisms to create code comprising instructions thatcan be executed directly, or through interpretation, micro-codeexecution, and the like, by one or more computer central processingunits (CPUs), Graphics Processing Units (GPUs), and the like.

The instructions can be executed on various types of computers orcomponents thereof, including, for example, personal computers, tabletcomputers, servers, smartphones, gaming devices, internet of thingsdevices, and the like.

The components shown in FIG. 21 for computer system (2100) are exemplaryin nature and are not intended to suggest any limitation as to the scopeof use or functionality of the computer software implementingembodiments of the present disclosure. Neither should the configurationof components be interpreted as having any dependency or requirementrelating to any one or combination of components illustrated in theexemplary embodiment of a computer system (2100).

Computer system (2100) may include certain human interface inputdevices. Such a human interface input device may be responsive to inputby one or more human users through, for example, tactile input (such as:keystrokes, swipes, data glove movements), audio input (such as: voice,clapping), visual input (such as: gestures), olfactory input (notdepicted). The human interface devices can also be used to capturecertain media not necessarily directly related to conscious input by ahuman, such as audio (such as: speech, music, ambient sound), images(such as: scanned images, photographic images obtain from a still imagecamera), video (such as two-dimensional video, three-dimensional videoincluding stereoscopic video).

Input human interface devices may include one or more of (only one ofeach depicted): keyboard (2101), mouse (2102), trackpad (2103), touchscreen (2110), data-glove (not shown), joystick (2105), microphone(2106), scanner (2107), camera (2108).

Computer system (2100) may also include certain human interface outputdevices. Such human interface output devices may be stimulating thesenses of one or more human users through, for example, tactile output,sound, light, and smell/taste. Such human interface output devices mayinclude tactile output devices (for example tactile feedback by thetouch-screen (2110), data-glove (not shown), or joystick (2105), butthere can also be tactile feedback devices that do not serve as inputdevices), audio output devices (such as: speakers (2109), headphones(not depicted)), visual output devices (such as screens (2110) toinclude CRT screens, LCD screens, plasma screens, OLED screens, eachwith or without touch-screen input capability, each with or withouttactile feedback capability−some of which may be capable to output twodimensional visual output or more than three dimensional output throughmeans such as stereographic output; virtual-reality glasses (notdepicted), holographic displays and smoke tanks (not depicted)), andprinters (not depicted).

Computer system (2100) can also include human accessible storage devicesand their associated media such as optical media including CD/DVD ROM/RW(2120) with CD/DVD or the like media (2121), thumb-drive (2122),removable hard drive or solid state drive (2123), legacy magnetic mediasuch as tape and floppy disc (not depicted), specialized ROM/ASIC/PLDbased devices such as security dongles (not depicted), and the like.

Those skilled in the art should also understand that term “computerreadable media” as used in connection with the presently disclosedsubject matter does not encompass transmission media, carrier waves, orother transitory signals.

Computer system (2100) can also include an interface to one or morecommunication networks. Networks can for example be wireless, wireline,optical. Networks can further be local, wide-area, metropolitan,vehicular and industrial, real-time, delay-tolerant, and so on. Examplesof networks include local area networks such as Ethernet, wireless LANs,cellular networks to include GSM, 3G, 4G, 5G, LTE and the like, TVwireline or wireless wide area digital networks to include cable TV,satellite TV, and terrestrial broadcast TV, vehicular and industrial toinclude CANBus, and so forth. Certain networks commonly require externalnetwork interface adapters that attached to certain general purpose dataports or peripheral buses (2149) (such as, for example USB ports of thecomputer system (2100)); others are commonly integrated into the core ofthe computer system (2100) by attachment to a system bus as describedbelow (for example Ethernet interface into a PC computer system orcellular network interface into a smartphone computer system). Using anyof these networks, computer system (2100) can communicate with otherentities. Such communication can be uni-directional, receive only (forexample, broadcast TV), uni-directional send-only (for example CANbus tocertain CANbus devices), or bi-directional, for example to othercomputer systems using local or wide area digital networks. Certainprotocols and protocol stacks can be used on each of those networks andnetwork interfaces as described above.

Aforementioned human interface devices, human-accessible storagedevices, and network interfaces can be attached to a core (2140) of thecomputer system (2100).

The core (2140) can include one or more Central Processing Units (CPU)(2141), Graphics Processing Units (GPU) (2142), specialized programmableprocessing units in the form of Field Programmable Gate Areas (FPGA)(2143), hardware accelerators for certain tasks (2144), and so forth.These devices, along with Read-only memory (ROM) (2145), Random-accessmemory (2146), internal mass storage such as internal non-useraccessible hard drives, SSDs, and the like (2147), may be connectedthrough a system bus (2148). In some computer systems, the system bus(2148) can be accessible in the form of one or more physical plugs toenable extensions by additional CPUs, GPU, and the like. The peripheraldevices can be attached either directly to the core's system bus (2148),or through a peripheral bus (2149). Architectures for a peripheral businclude PCI, USB, and the like.

CPUs (2141), GPUs (2142), FPGAs (2143), and accelerators (2144) canexecute certain instructions that, in combination, can make up theaforementioned computer code. That computer code can be stored in ROM(2145) or RAM (2146). Transitional data can be also be stored in RAM(2146), whereas permanent data can be stored for example, in theinternal mass storage (2147). Fast storage and retrieve to any of thememory devices can be enabled through the use of cache memory, that canbe closely associated with one or more CPU (2141), GPU (2142), massstorage (2147), ROM (2145), RAM (2146), and the like.

The computer readable media can have computer code thereon forperforming various computer-implemented operations. The media andcomputer code can be those specially designed and constructed for thepurposes of the present disclosure, or they can be of the kind wellknown and available to those having skill in the computer software arts.

As an example and not by way of limitation, the computer system havingarchitecture (2100), and specifically the core (2140) can providefunctionality as a result of processor(s) (including CPUs, GPUs, FPGA,accelerators, and the like) executing software embodied in one or moretangible, computer-readable media. Such computer-readable media can bemedia associated with user-accessible mass storage as introduced above,as well as certain storage of the core (2140) that are of non-transitorynature, such as core-internal mass storage (2147) or ROM (2145). Thesoftware implementing various embodiments of the present disclosure canbe stored in such devices and executed by core (2140). Acomputer-readable medium can include one or more memory devices orchips, according to particular needs. The software can cause the core(2140) and specifically the processors therein (including CPU, GPU,FPGA, and the like) to execute particular processes or particular partsof particular processes described herein, including defining datastructures stored in RAM (2146) and modifying such data structuresaccording to the processes defined by the software. In addition or as analternative, the computer system can provide functionality as a resultof logic hardwired or otherwise embodied in a circuit (for example:accelerator (2144)), which can operate in place of or together withsoftware to execute particular processes or particular parts ofparticular processes described herein. Reference to software canencompass logic, and vice versa, where appropriate. Reference to acomputer-readable media can encompass a circuit (such as an integratedcircuit (IC)) storing software for execution, a circuit embodying logicfor execution, or both, where appropriate. The present disclosureencompasses any suitable combination of hardware and software.

While this disclosure has described several exemplary embodiments, thereare alterations, permutations, and various substitute equivalents, whichfall within the scope of the disclosure. It will thus be appreciatedthat those skilled in the art will be able to devise numerous systemsand methods which, although not explicitly shown or described herein,embody the principles of the disclosure and are thus within the spiritand scope thereof.

APPENDIX A: ACRONYMS

JEM: joint exploration modelVVC: versatile video codingBMS: benchmark set

MV: Motion Vector HEVC: High Efficiency Video Coding SEI: SupplementaryEnhancement Information VUI: Video Usability Information GOPs: Groups ofPictures TUs: Transform Units, PUs: Prediction Units CTUs: Coding TreeUnits CTBs: Coding Tree Blocks PBs: Prediction Blocks HRD: HypotheticalReference Decoder SNR: Signal Noise Ratio CPUs: Central Processing UnitsGPUs: Graphics Processing Units CRT: Cathode Ray Tube LCD:Liquid-Crystal Display OLED: Organic Light-Emitting Diode CD: CompactDisc DVD: Digital Video Disc ROM: Read-Only Memory RAM: Random AccessMemory ASIC: Application-Specific Integrated Circuit PLD: ProgrammableLogic Device LAN: Local Area Network

GSM: Global System for Mobile communications

LTE: Long-Term Evolution CANBus: Controller Area Network Bus USB:Universal Serial Bus PCI: Peripheral Component Interconnect FPGA: FieldProgrammable Gate Areas

SSD: solid-state drive

IC: Integrated Circuit CU: Coding Unit TSM: Transform Skip Mode IBC:Intra Block Copy

DPCM: Differential pulse-code modulationBDPCM: Block based DPCM

What is claimed is:
 1. A method for video decoding in a decoder,comprising: determining, from a coded video bitstream, a portion of ablock is subject to a zero-out operation based on a threshold number ofsamples being less than a number of samples in one of a height dimensionor a width dimension of the block; determining whether a mixture of alossless coding mode and one or more lossy coding modes is allowed inthe coded video bitstream; and reconstructing, by a processor, the blockbased on the one or more lossy coding modes in response to (i) themixture of the lossless coding mode and the one or more lossy codingmodes being allowed in the coded video bitstream and (ii) the portion ofthe block being subject to the zero-out operation.
 2. The method ofclaim 1, wherein the determining whether the mixture of the losslesscoding mode and the one or more lossy coding modes is allowed comprises:decoding, from the coded video bitstream, a flag that indicates whetherthe coded video bitstream is only coded in the lossless coding mode. 3.The method of claim 2, wherein the flag is signaled in at least one of avideo parameter set (VPS), a sequence parameter set (SPS), a pictureparameter set (PPS), a slice header, a tile header, and a tile groupheader.
 4. The method of claim 2, further comprising: partitioning theblock into multiple blocks based on (i) the flag indicating the codedvideo bitstream is only coded in the lossless coding mode and (ii) theportion of the block being subject to the zero-out operation, whereinthe reconstructing includes reconstructing the multiple blocksrespectively from the coded video bitstream based on the lossless codingmode.
 5. The method of claim 4, wherein the partitioning includes atleast one of: splitting the block into a left block and a right block ofequal sizes based on the number of samples in the width dimension of theblock being equal to or greater than the threshold number of samples andthe number of samples in the height dimension of the block being lessthan the threshold number of samples; splitting the block into a topblock and a bottom block of equal sizes based on the number of samplesin the width dimension of the block being less than the threshold numberof samples and the number of samples in the height dimension of theblock being equal to or greater than the threshold number of samples;and splitting the block into a top left block, a top right block, abottom left block, and a bottom right block of equal sizes based on thenumber of samples in both of the width dimension and the heightdimension of the block being equal to or greater than the thresholdnumber of samples.
 6. The method of claim 1, wherein the block is atleast one of a coding block and a transform block.
 7. The method ofclaim 1, further comprising: setting a maximum coding tree unit (CTU)size to be less than the threshold number of samples to avoid thezero-out operation in coding tree units.
 8. The method of claim 2,wherein the flag is a first flag associated with the block and indicatesthe block is only coded in the lossless coding mode, the first flagbeing different from a second flag for indicating a bypass of transformand quantization operations on the block.
 9. The method of claim 2,wherein the flag is associated with the block and indicates the block isonly coded in the lossless coding mode, which includes a bypass oftransform and quantization operations on the block.
 10. The method ofclaim 2, wherein the flag indicates whether at least one of a jointchroma residual mode coding tool, a luma mapping with chroma scaling(LMCS) coding tool, a residual differential pulse code modulation(RDPCM) coding tool, a multiple transform selection (MTS) coding tool, atransform skip mode (TSM) coding tool, a sub-block transform (SBT)coding tool, a secondary transform coding tool, a sample adaptive offset(SAO) coding tool, a deblocking coding tool, or an adaptive loop filter(ALF) coding tool is enabled.
 11. An apparatus for video decoding,comprising: processing circuitry configured to: determine, from a codedvideo bitstream, a portion of a block is subject to a zero-out operationbased on a threshold number of samples being less than a number ofsamples in one of a height dimension or a width dimension of the block;determine whether a mixture of a lossless coding mode and one or morelossy coding modes is allowed in the coded video bitstream; andreconstruct the block based on the one or more lossy coding modes inresponse to (i) the mixture of the lossless coding mode and the one ormore lossy coding modes being allowed in the coded video bitstream and(ii) the portion of the block being subject to the zero-out operation.12. The apparatus of claim 11, wherein the processing circuitry isconfigured to: decode, from the coded video bitstream, a flag thatindicates whether the coded video bitstream is only coded in thelossless coding mode.
 13. The apparatus of claim 12, wherein the flag issignaled in at least one of a video parameter set (VPS), a sequenceparameter set (SPS), a picture parameter set (PPS), a slice header, atile header, and a tile group header.
 14. The apparatus of claim 12,wherein the processing circuitry is configured to: partition the blockinto multiple blocks based on (i) the flag indicating the coded videobitstream is only coded in the lossless coding mode and (ii) the portionof the block being subject to the zero-out operation; and reconstructthe multiple blocks respectively from the coded video bitstream based onthe lossless coding mode.
 15. The apparatus of claim 11, wherein theblock is at least one of a coding block and a transform block.
 16. Theapparatus of claim 11, wherein the processing circuitry is configuredto: set a maximum coding tree unit (CTU) size to be less than thethreshold number of samples to avoid the zero-out operation in codingtree units.
 17. The apparatus of claim 12, wherein the flag is a firstflag associated with the block and indicates the block is only coded inthe lossless coding mode, the first flag being different from a secondflag for indicating a bypass of transform and quantization operations onthe block.
 18. The apparatus of claim 12, wherein the flag is associatedwith the block and indicates the block is only coded in the losslesscoding mode, which includes a bypass of transform and quantizationoperations on the block.
 19. The apparatus of claim 12, wherein the flagindicates whether at least one of a joint chroma residual mode codingtool, a luma mapping with chroma scaling (LMCS) coding tool, a residualdifferential pulse code modulation (RDPCM) coding tool, a multipletransform selection (MTS) coding tool, a transform skip mode (TSM)coding tool, a sub-block transform (SBT) coding tool, a secondarytransform coding tool, a sample adaptive offset (SAO) coding tool, adeblocking coding tool, or an adaptive loop filter (ALF) coding tool isenabled.
 20. A non-transitory computer-readable medium storinginstructions which, when executed by a computer, cause the computer toperform: determining, from a coded video bitstream, a portion of a blockis subject to a zero-out operation based on a threshold number ofsamples being less than a number of samples in one of a height dimensionor a width dimension of the block; determining whether a mixture of alossless coding mode and one or more lossy coding modes is allowed inthe coded video bitstream; and reconstructing the block based on the oneor more lossy coding modes in response to (i) the mixture of thelossless coding mode and the one or more lossy coding modes beingallowed in the coded video bitstream and (ii) the portion of the blockbeing subject to the zero-out operation.